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The  Ada  language  Is  not  just  another  progranwilng 
language;  Ada  will  be  Installed  with  a  pre¬ 
defined  environment  that  will  provide  pro- 
graitwers  with  a  standard  set  of  tools  to  perform 
their  jobs.  In  addition  to  tools  requisite  for 
any  environment  (1.  e.,  a  compiler,  an  editor, 
etc.),  tools  may  be  provided  for  data  collec¬ 
tion.  These  data  will  provide  feedback  to  the 
project’s  managers  and  programmers  during  the 
software  development  process  and  will  predict 
the  operational  characteristics  of  the  system 
under  development.  In  the  past.  Instances  of 
data  collection  efforts  on  software  projects 
have  been  erratic.  However,  with  the  commonal¬ 
ity  that  will  now  occur  for  Ada  environments, 
there  Is  an  opportunity  to  Introduce  systematic 
methods  and  procedures  for  measurement  on  a  wide 
scale.  Thus  there  Is  a  need  for  the  selection  of 
metrics  that  are  relevant  and  useful  in  an  Ada 
environment. 

Different  metrics  are  useful  for  different 
purposes  and  at  different  times  In  the  life 
cycle  of  a  software  development  project.  For 
example,  project  managers  need  measures  to  pre¬ 
dict  resources  and  schedules,  to  track  costs  and 
to  locate  potential  problem  areas.  These  needs 
differ  from  those  of  a  customer  or  end  user  who 
would  like  some  measure  of  the  reliability  of 
the  system  and  proof  of  the  thoroughness  of  the 
testing  that  has  been  done.  Still  different  are 
the  needs  of  a  programmer  who  would  benefit  from 
feedback  about  the  complexity  of  a  just-compiled 
module.  The  problem  of  defining  a  unified  set 
of  metrics  Is  not  simple  and  straightforward. 
He  must  consider  the  needs  of  a  variety  of 
people  who  come  Into  contact  with  a  system  over 
a  long  period  of  time., 

Seven  researchers  from  the  University  of 
Maryland*  and  from  General  Electric  have  joined 
In  an  eighteen-month  collaborative  effort  to 
study  this  problem.  Part  of  the  effort  has  In¬ 
volved  collecting  measurements  from  an  on-going 
Ada  software  development  project.  Our  general 
approach  has  been  to  collect  a  great  deal  of 
Information,  to  try  to  evaluate  the  different 
types  of  measures,  and  to  select  those  that 
appear  most  useful.  This  paper  will  describe 
the  software  project  selected,  the  data  collec¬ 
tion  effort  and  the  candidate  metrics  that  are 
being  considered. 


•Members  of  the  University  of  Maryland  team  are: 
Victor  Basin,  John  Gannon,  Elizabeth  Katz,  and 
Marvin  Zelkowitz. 


He  chose  a  realistically  large  and  complex  soft¬ 
ware  development  project  for  study.  It  Involved 
re-implementation  of  a  portion  of  a  working 
ground  support  system  for  a  communication  satel¬ 
lite.'  The  original  system  was  developed  by 
General  Electric  and  consisted  of  approximately 
100,000  lines  of  FORTRAN  and  assembly  code.  A 
subset  of  the  original  system  was  selected  for 
redesign  and  Implementation  In  Ada.  This  subset 
Is  executable  apart  from  the  larger  system.  It 
Includes  functions  to  receive  an  operator's 
Inputs,  to  perform  complex  computations  and  to 
display  output  graphically.  Also  included  are 
several  concurrent  processes  to  monitor  and 
display  telemetry  data  from  the  satellite. 

The  project  began  In  February  1982  with  a 
month  of  formal  training  in  Ada.  Following 
this,  the  lead  programmer  and  back-up  programmer 
produced  a  requirements  document  describing  the 
subsystem.  The  design  was  developed  In  an 
Ada-like  program  design  language.  Coding  began 
In  August  and  was  completed  In  November,  1982. 
Approximately  10,000  lines  of  Ada  source  code. 
Including  comments,  were  produced.  Some 

compilation  has  been  done  using  the  NYU  Ada/Ed 
Interpreter.  Testing  will  be  done  as  soon  as  a 
full  Ada  compiler  Is  available. 

Our  approach  to  measurement  for  this  project  has 
been  systematic  and  thorough.  He  began  by 

defining  goals  for  our  data-collectlon  effort. 
The  goals  fell  naturally  Into  three  categories: 
those  relating  to  software  development  projects 
In  general,  those  relating  to  the  use  of  Ada  as 
a  design  and  implementation  language,  .and  those 
relating  to  metrics  for  the  APSE,  the  Ada 
Prograwalng  Support  Environment.  Following 

that,  a  number  of  specific  questions  related  to 
each  goal  were  developed.  The  goals  and  the 
questions  relevant  to  each  are  listed  in  the 
Appendix. 


Three  methods  of  data  collection  were  chosen  to 
answer  the  questions  associated  with  the  goals. 
First,  eight  different  data  collection  forms 
were  adapted  from  those  developed  by  Victor 
Basin  and  Marvin  Zelkowitz  for  the  NASA  Soft¬ 
ware  Engineering  Laboratory. (1 )  The  forms  were 
designed  to  be  completed  by  the  members  of  the 
development  team.  The  data  collected  on  the 
forms  provides  a  complete  record  of  activities 
during  the  development  process.  The  forms  focus 
on  three  types  of  data:  effort,  changes  and 
errors.  These  data  will  be  discussed  In  detail 
below. 
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Second,  an  on-Hne  procedure  was  developed  for 
recording  all  versions  of  the  design  and  the 
code  for  later  analysis  by  a  YACC-generated 
processor.  The  processor  provides  a  static  view 
of  the  system  by  counting  such  data  as  the  types 
of  Ada  features  used  In  each  module,  the  data 
exchanged  between  modules,  etc. 

Third,  the  members  of  the  progranmlng  team  were 
Interviewed  after  training  and  at  the  end  of 
each  major  phase  of  the  development  cycle  to 
determine  what  new  concepts  they  had  learned, 
what  were  their  attitudes  about  Ada,  and  how 
those  attitudes  were  changing. 

In  order  to  assure  the  quality  of  the  data 
collected,  the  research  team  was  diligent  about 
cross-checking  the  data  for  accuracy.  The 
results  of  this  activity  were  surprising  to 
researchers  who  had  previously  been  Involved  In 
laboratory  experiments.  Techniques  for 
controlled  experimentation  are  well  developed, 
and  on-line  programming  experiments  can  be  used 
to  collect  extremely  precise  time  and  error 
data.  However,  In  this  field  study,  we  found  a 
very  different  situation.  Athough  the  program¬ 
ming  team  understood  the  necessity  for  recording 
the  Information  precisely,  there  were  wide 
variations  In  estimates.  For  example,  one 
programmer  recorded  a  time  of  one  hour  for  a 
design  walkthrough  and  another  recorded  a  time 
of  three  hours  for  the  same  walkthrough, 
although  each  person  attended  the  whole 
session.  An  Inquiry  revealed  that  the  walk¬ 
through  on  the  design  had  been  completed  In  an 
hour,  but  the  group  had  remained  together  for  an 
additional  two  hours  In  order  to  discuss 
methodology.  Thus  It  was  necessary  to  provide 
constant  monitoring  of  the  data  collection 
effort  at  the  site  of  the  programming  activity. 
In  fact,  all  team  meetings  were  attended  by  at 
least  one  member  of  the  research  team. 

Collecting  data  with  an  on-line  system  would 
have  relieved  some  of  the  problems  we  found  with 
manual  data  collection  procedures.  Incomplete 
or  Inconsistent  data  values  would  have  been  more 
readily  apparent.  Further,  on-line  entry  of 
data  might  have  seemed  more  palatable  to  the 

progratiwlng  team.  The  extensive  data  collection 
effort  done  here  Interfered  to  some  degree  with 
their  progress  of  the  project:  filling  out  the 
forms  required  time  and  effort.  On-line  data 
collection  might  have  been  less  apparent  to  the 
programming  team,  even  though  the  same  data 
would  have  been  obtained.  Finally,  such  a 
system  would  eliminate  the  need  for  later  entry 
of  the  data  Into  the  data  base.  In  our  selec¬ 
tion  of  metrics,  we  are  considering  the  ease 
with  which  the  data  for  the  metrics  can  be 

collected  and  the  degree  to  which  It  can  be 
obtained  unobtrusively. 

A  large  number  of  candidate  metrics  have  been 
defined  and  investigated.  All  of  those  that 
have  a  possibility  of  being  useful  are  being 

evaluated.  Area  C,  as  shown  In  the  Appendix, 

shows  questions  specifically  related  to  defining 


metrics  for  the  APSE.  Some  of  these  metrics  may 
be  useful  in  any  software  development  environ¬ 
ment,  regardless  of  the  design  or  implementation 
language  used;  others  may  be  useful  only  for  the 
Ada  environment.  Some  metrics  may  provide 
enough  Information  to  make  other  measures 
superfluous.  Two  or  more  may  reflect  the  same 
Information,  but  the  data  for  one  may  be  more 
difficult  to  collect.  Some  metrics  may  not 
contribute  useful  Information  at  all.  Thus  we 
are  comparing  the  metrics  In  order  to  discard 
those  that  are  not  needed  and  to  select  a  group 
that  will  be  truly  useful. 

Data  collection  for  this  project  has  centered 
largely  on  two  kinds  of  metrics,  those  dealing 
with  the  software  development  process  Itself  and 
those  dealing  with  the  evolving  product.  Data 
collection  for  the  process  metrics  Include:  a) 
the  effort  expended,  b)  the  changes  to  the 
design  or  code,  and  c)  the  errors  that  occur. 

For  effort  data,  we  collected  the  number  of 
hours  spent  In  various  activities  during  the 
life  cycle.  He  also  recorded  the  names  of  the 
modules  and  the  types  of  activity  (e.  g.,  design 
review  or  code  reading)  on  which  the  time  was 
expended.  Because  the  human  effort  required  to 
complete  a  project  generally  accounts  for  its 
largest  single  cost,  effort  data  can  be  used  to 
predict  costs  of  future  projects  with  similar 
parameters  (e.  g.,  size  and  application). 

Effort  metrics  are  also  useful  for  measuring 
productivity  and  for  assessing  the  impact  of  new 
tools  and  techniques  on  productivity. 

Every  software  development  will  undergo  changes 
to  previous  documents  (whether  anticipated  or 
not).  Some  development  techniques,  such  as 
Iterative  enhancement,  specifically  assume  that 
continual  evolution  Is  desirable.  Others  assume 
there  will  be  a  single  pass  through  each  phase 
with  any  further  ..langes  being  carefully 
controlled.  In  either  case.  Information  about 
changes  provides  Important  Insights  into  the 
project.  Data  collected  about  a  change  to  our 
system  Included:  the  reason  for  the  change,  the 
time  spent  making  the  change,  additional  docu¬ 
ments  examined  during  the  change,  and  changes  to 
other  documents  because  of  the  change.  These 
data  help  determine  the  costs  of  various  types 
of  changes  and  assist  in  predicting  the  modifia¬ 
bility  of  the  system.  Change  data  are  also 
useful  for  evaluating  whether  the  development 
methodology  Is  successful  for  the  environment. 
For  example,  large  numbers  of  major  changes 
might  Indicate  that  a  different  development 
methodology  should  be  used.  In  such  a  case,  an 
Incremental  development  approach  might  be  bene¬ 
ficial  In  establishing  the  desirable  features  of 
the  system.  (2) 

Data  about  errors  Included  a  description  of  the 
error,  the  activities  used  to  detect  the  error, 
the  time  at  which  the  error  entered  the  system, 
and  an  evaluation  of  whether  the  error  was 
related  to  the  Ada  progranmlng  language.  These 
data  provide  Information  about  the  quality  of 
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the  product.  We  can  also  use  error  data  to 
Improve  our  methodology.  If  we  know  which  types 
of  errors  are  more  difficult  to  correct,  we  can 
put  an  emphasis  on  trying  to  locate  and  correct 
those  types  of  errors  during  walkthroughs  or 
other  reviews  of  the  product.  For  errors 
related  to  the  use  of  Ada,  error  data  can  be 
used  to  determine  where  emphasis  Is  needed  In 
future  training  courses. 

Product  metrics  are  the  other  major  type  of 
metric  we  have  been  examining.  They  deal  with 
the  characteristics  of  the  software  Itself  and 
can  be  divided  Into  t*«  categories:  static  and 
dynamic.  Static  metrics  can  be  collected  from 
the  software  at  any  point  In  Its  development. 
Dynamic  metrics  are  run-time  measures.  There 
are  a  myriad  of  measurements  that  can  be 
collected  for  both  categories;  only  a  few 
representative  ones  are  discussed  below. 

Static  metrics  are  useful  for  determining  the 
complexity  and  quality  of  the  code.  Collecting 
such  metrics  at  several  phases  of  the  life  cycle 
provides  a  view  of  the  way  in  which  the  system 
changes  across  the  phases.  Static  metrics  can 
be  subdivided  Into  three  types;  site,  control 
and  data  metrics.  Size  metrics  are  Indicators 
of  the  volume  of  a  product  and  the  amount  of 
work  performed.  They  correlate  well  with  effort 
and  are  used  for  estimating  costs,  comparing 
products  and  measuring  productivity. (3)  Lines 
of  code  are  frequently  used  to  measure  size. 
However,  there  are  alternate  ways  to  count  lines 
of  code.  Comment  lines  may  be  Included  or 
omitted,  depending  on  the  use  of  the  metric.  If 
we  are  measuring  productivity,  we  probably  want 
to  Include  both  code  and  cooment  lines. 
However,  the  sel  f-doctanentlng  features  of  the 
Ada  syntax  may  produce  a  smaller  ratio  of 
comment  lines  to  code  lines  than  Is  usually 
considered  good  programalng  practice  In  other 
languages.  Further  difficulties  may  arise  when 
computing  the  size  of  a  system  which  Incorpor¬ 
ates  previously  written  components  or  Is  a 
modification  of  a  previously  developed  system. 
Should  productivity  measures  include  only  new 
development  work  or  should  they  reflect  all  of 
the  code  that  Is  delivered?  Certain  features  of 
the  Ada  language  (1.  e.,  packages,  generics) 
will  make  possible  large  libraries  of  reusable 
components  that  can  be  Incorporated  Into  a  new 
system  much  the  way  In  which  circuits  are  now 
constructed  with  off-the-shelf  chips.  This  new 
trend  In  software  design,  as  well  as  the  self- 
documenting  capabilities  of  the  syntax,  will 
necessitate  fresh  approaches  to  size  measures 
which  are  meaningful  for  Ada. 

Control  flow  metrics  measure  the  complexity  of  a 
product.  McCabe's  v^G)  Is  a  count  of  the 
number  of  basic  controT  path  se^aents  In  a 
computer  program. (4)  This  value  depends  on  the 
number  of  decision  nodes  and  the  branches 
emanating  from  those  nodes.  V(G)  was  originally 
developed  as  part  of  a  strategy  for  testing 
software.  However,  It  has  been  suggested  that 
v(G)  Is  also  useful  as  a  measure  of  psycho¬ 


logical  complex1ty.(5)  The  theory  1s  that  a 
program  with  many  decisions  Is  psychologically 
complex:  the  more  decisions,  the  more  difficult 
a  program  Is  to  understand  and  modify.  One 
exception  to  this  Is  a  case  statement  where 
there  Is  one  path  for  each  of  several  similar 
choices.  McCabe  suggests  limiting  v(G)  to  10 
for  any  module  except  where  v(G)  Is  Infl ated  by 
a  large  case  statement. 


The  current  method  of  counting  v(Gl  may  not  be 
sufficient  for  the  Ada  language.  STa’s  explicit 
structures  for  exception  handling  alter  the 
control  flow  normally  found  in  structured 
languages.  Additional  paths  can  be  explicitly 
generated  to  handle  situations  that  would  cause 
faulty  execution  In  other  languages.  For 
example.  In  some  other  languages,  an  attempt  to 
read  past  an  end-of-flle  marker  would  produce  an 
error  message  and  would  terminate  the  job 
stream.  In  Ada  It  Is  possible  to  specify  the 
means  to  handle  this  possibility  and  to  continue 
processing.  This  exception  handling,  however, 
alters  the  normal  flow  of  control  of  the 
program,  and  It  Is  necessary  to  consider  alter¬ 
natives  for  calculating  v(G).  One  method  would 
Ignore  the  potential  occurrence  of  exceptions. 
A  second  method  would  account  for  all  possible 
oaths  of  execution  as  the  result  of  the  occur¬ 
rence  of  an  exception.  The  first  alternative 
alters  the  premise  that  all  basic  paths  through 
the  module  are  being  counted  and  tested.  The 
second  alternative  Is  theoretically  more 
appealing,  but  It  would  greatly  increase  the 
number  of  paths  and  would  thus  make  testing  very 
difficult. 

Data  metrics  analyze  the  organization  of  the 
data  structures  within  and  between  modules  In  an 
attempt  to  measure  the  ease  with  which  they  can 
be  understood  and  modified.  Data  metrics  of 
Interest  Include  the  number  of  average  live 
variables  per  statement,  the  percentage  of 
global  variables,  and  the  number  of  programmer- 
defined  types.  Some  data  metrics,  such  as  the 
Information  flow  metrics  of  Henry  and  Kafura, 
focus  on  the  Interconnections  between  system 

components.  (6,  71  Myers  has  Indicated  that 

Interfaces  are  Important  because  many  serious, 
hard-to-find  errors  In  systems  result  from  a 
lack  of  understanding  of  module  inter¬ 
dependencies  and  from  changes  made  to  global 
data  areas.  (8) 

More  work  Is  also  needed  In  the  area  of  data 
metrics  for  Ada.  Is  the  density  of  data  flow 
across  modules  as  proposed  by  Henry  and  Kafura  a 
useful  metric  for  the  Ada  language?  The  Ada 
block  structure,  visibility  rules  and  packaging 
features  all  affect  the  way  In  which  data  is 
apportioned  and  the  degree  to  which  data  Is  or 
Is  not  visible  at  various  locations  In  the 
code.  Thus  there  Is  a  need  to  define  how  to 
measure  the  number  of  global  and  local  data 
structures  and  how  to  measure  the  quality  with 
which  the  data  structures  are  encapsulated  In 

packages.  These  measures  should  be  helpful  In 


rteterminlnq  whether  the  encapsulation  makes  the 
best  use  of  Ada's  features  for  producing 
maintainable,  reuseable  software. 

Dynamic  metrics  provide  another  method  for 
measuring  the  software  product.  Dynamic 
measures  can  be  divided  Into  execution  and  test 
coverage  metrics.  Execution  metrics  are  useful 
for  tuning  a  system  to  make  It  run  efficiently. 
Execution  metrics  Include  the  number  of  times 
each  statement  Is  executed  and  the  CPU  time  used. 

Test  coverage  metrics  help  determine  the  degree 
and  quality  of  the  testing  that  has  been  done. 
McCabe's  metric  has  already  been  discussed. 
Other  candidate  test  coverage  metrics  Include 
the  number  of  statements  executed  and  the  number 
of  branches  executed.  Because  Ada  has  mecha¬ 
nisms  for  concurrent  processing,  metrics  are 
needed  to  measure  tasking  features  and  usage. 
Further,  concurrent  processing  presents  oppor¬ 
tunities  for  both  starvation  and  deadlock; 
methods  are  needed  for  predicting  the 
possibility  of  these  occurrences  during  the 
testing  phase  of  the  life  cycle. 

The  analysis  of  the  data  from  this  project  and 
the  subsequent  selection  of  a  suggested  set  of 
metrics  for  an  Ada  environment  Is  currently 
underway.  The  systematic  approach  to  the 
goal-driven  data  collection  effort  will  provide 
a  beginning  methodology  for  subsequent  efforts 
for  monitoring  software  projects.  The  metrics 
selected  will  be  useful  for  all  of  the  differing 
needs  of  those  assxiated  with  an  Ada  system 
throughout  the  life  cycle.  If  we  can  automate 
and  Insert  a  common  set  of  metrics  Into  the 
APSE,  data  collection  may  become  an  Integral 
part  of  software  methodology,  and  comparisons 
across  Ada  projects  of  all  types  will  become 
feasible.  Further,  quantitative  Indices  of  the 
progress  of  a  project  will  be  available  for 
managers,  procurement  officers,  designers  and 
others  with  a  need  for  such  Information. 


APPENDIX 


Area  A:  GcDcrie  goals  for  any  software 
development  project 

Goal  Al:  Characterize  the  effort  in  the  project. 

II  How  was  the  effort  distributed  over  the 
phases  of  the  project? 

21  How  was  the  effort  for  the  project 
distributed  over  time? 

31  How  was  the  effort  distributed  across 
different  functions  in  the  software? 

41  How  are  the  error  distributions  similar 
to  or  different  from  other  comparable 
software  developments? 

Goal  A2:  Characterize  the  changes. 

11  How  are  the  changes  to  the  system 
distributed  over  the  software  develop¬ 
ment  cycle? 

21  How  is  the  time  for  handling  a  change 
distributed?  How  long  does  it  take  to 
design  and  implement  the  change? 


31  Are  certain  features  of  .Ada  or  certain 
types  of  errors  associated  with 
particular  programmers?  Why? 

41  Do  certain  programmers  have  problems 
with  certain  aspects  of  the  language? 

3)  Do  programmers  want  to  use  features 
available  in  other  languages  that  are  not 
available  in  .Ada? 

6)  Are  some  features  of  the  language 
overused,  used  incorrectly,  or  us^ 
inappropriately  in  the  programmers 
enthusiasm  to  use  what  they  have 
learned? 

7)  Do  people  with  no  previous  high  level 
language  experience  have  more  or  fewer 
problems  with  Ada  than  people  with 
high  level  language  experience? 


Area  B:  Goals  relating  to  Ada  as  a  design  and 
implementation  language 

Goal  Bl:  Characterize  the  errors  made. 

11  How  were  the  errors  found?  le.g..  design 
review,  inspection  of  output,  etc.) 

21  What  were  the  non-Ada  causes  of  the 
errors?  le.g..  requirements  misinter¬ 
preted.  mistake  in  computation,  etc.i 

31  What  features  of  Ada  are  commonly 
involved  in  errors? 

41  Are  there  features  of  Ada  that  cause 
problems  when  they  ve  used  together'’ 

51  Are  errors  attribute  to  confusion  with 
another  lan^age?  to  a  lack  of 
understanding  of  Ada?  to  a  lack  of 
experience  with  a  feature? 

6l  .Are  the  errors  made  when  using  .Ada  as 
a  design  language  different  than  those 
made  when  coding? 

71  Where  was  the  information  found  that 
was  needed  to  correct  the  error?  le  g.. 
.Ada  Reference  Manual,  another 
programmer,  etc.i 

8i  Is  the  error  characteristic  of  the  feature 
or  of  the  particular  application  it 
involved? 

Goal  B2:  Determine  whether  certain  aspects  of 
Ada  are  difficult  to  use  for  certain 
applications. 

11  Are  there  certain  aspects  of  .Ada  that  do 
not  apply  to  this  type  of  project? 

21  Are  there  techniques  usually  used  for 
this  type  of  application  that  are  difficult 
to  implement  in  Ada? 

Goal  B3:  Determine  which  aspects  of  .Ada 

contribute  positively  to  the  design  and 
programming  environment. 

1 1  Are  errors  easy  to  find?  to  correct? 

21  Is  there  a  large  amount  of  parallel 
development  once  the  interfaces  are 
defined? 

31  How  effective  is  Ada  in  reduang 
mierface  errors?  producing  software 
that  IS  easy  to  change?  reducing  the 
development  effort,  especially  in 
realtime  problems'’ 

Goal  B4:  Determine  which  combinations  of  .Ada  s 
features  are  naturally  used  together. 

II  How  fully  IS  the  language  used'’ 


2)  Are  there  certain  features  of  Ada  that 
are  avoided  because  they  are  difficult  to 
learn?  difficult  to  use?  poorly 
implemented?  error  prone? 

Goal  BS:  Determine  the  effect  of  using  Ada  as  a 
PDL. 

ll  Does  Ada  PDL  allow  sufficient  abstrac¬ 
tion  at  the  early  stages  of  design? 

2)  Is  the  language  really  being  used  as  a 
design  language? 

31  Does  the  use  of  Ada  PDL  cause  a 
preoccupation  with  syntax  during  the 
design  stage? 

41  What  is  the  expansion  of  Ada  PDL  to 
code? 

51  Does  Ada  PDL  guide  the  design  of  the 
project  or  are  portions  of  the  system 
primarily  other  language  programs 
written  in  Ada  syntax? 

61  Is  there  an  adequate  combination  of 
features  of  Ada  for  use  as  a  PDL? 

Tl  Are  the  most  expensive  errors  found 
while  using  a  particular  set  of  features 
of  Ada  as  a  PDL? 

8)  Are  errors  uncovered  at  the  design  stage 
that  ordinarily  would  have  been 
uncovered  during  coding  because  of  the 
use  of  Ada  PDL? 

9l  What  percentage  of  the  interface  errors 
are  uncovered  during  the  design  stage? 

Goal  B6:  Characterize  the  programmers  and 

associate  their  background  with  their 
use  of  Ada. 

II  What  are  the  programmers'  opinions  of 
Ada  before  they  tegin  this  project? 
during?  afterward? 

21  What  is  each  programmer  s  background 
with  other  languages? 

31  Is  there  a  relationship  between  how  far 
into  development  the  change  was  needed 
and  how  much  effort  was  spent  on  the 
change?  How  many  sections  it  affected? 

4)  What  kind  of  changes  were  made?  le.g.. 
error  correction,  planned  enhancement, 
etc.  I 

5 1  How  many  components  are  involved  in 

the  typical  change? 

61  How  many  changes  are  caused  by  a 
previous  change? 

7l  How  was  the  need  for  change 
deterimned? 

8l  How  many  and  what  kind  of  interface 
changes  need  to  be  made? 

Area  C  :  Goals  Relating  to  Metrics  for 

the  APSE 


Goal  Cl:  Select  a  set  of  static  (size, 

control  and  data)  metrics  for 
the  APSE 

1 1  .Are  [here  differences  in  the 
implications  of  various  counting 
measures?  .Are  some  measures  more 
useful  than  others? 

2)  Do  cenain  program  measures 

provide  enough  information  to  make 
other  measures  superfluous’ 


3)  Which  static  metrics  can  be  applied 
throughout  the  design  and  code 
phases.  Which  cannot? 

4)  VVhich  static  metrics  help  predict 
run-time  behavior  le.g..  reliabiiitv. 
etc. )? 

3)  Which  static  metrics  can  be 
measured  most  easilv’ 

Goal  Cl.l:  Develop  a  set  of  size  metrics  for 
the  APSE 

1)  What  size  metrics  best  predict  effort’ 

2)  What  serves  as  a  useful  size  metric 
(e.g..  lines  of  code,  modules!  in  .Ada.’ 

3)  What  constitutes  a  statement  in 
.Ada? 

4)  How  should  an  e.\ecutable  statement 
be  defined  in  .Ada? 

5)  What  features  of  .Ada  should  be 
grouped  when  counting  the  number 
of  times  certain  features  are  used? 

6)  How  useful  IS  Halstead's  software 
science  approach  with  .Ada.’ 

Goal  Cl. 2:  Develop  a  set  of  control  metrics 
for  the  .APSE 

1)  How  can  tasking  and  e.xceptions  be 
integrated  into  the  control  metrics’ 

2)  How  useful  IS  .McCabe's  cvclomatic 
comple.xitv  measure?  How  does  the 
cvclomatic  comple.xitv  compare  with 
the  essential  comple.xitv’ 

3)  How  useful  are  measures  of  nesting 
complexitv  and  depth? 

Goal  Cl. 3:  Develop  a  set  of  data  metrics 
for  the  .APSE 

1)  How  can  the  complexitv  of  data 
structures  be  measured’ 

2)  What  influences  the  number  of 
programmer  defined  tvpes’ 

3)  How  does  the  use  of  .Ada  influence 
the  number  of  inputs  to  and  outputs 
from  a  module? 

4)  How  does  .Ada  influence  the  use  of 
global  data.’ 

51  How  does  the  use  of  modules  affect 
the  treatment  of  data  within  a 
program? 

6)  How  should  the  span  of  a  variable 
be  measured?  Is  there  a  use  for  the 
span  information’ 

')  What  do  the  data  bindings  suggest 
about  the  structure  of  the  svstem’ 

8)  Does  the  densitv  of  the  data  tlow 
across  modules  provide  useful 
feedback  about  the  structure  of  the 
svstem?  i.e..  are  information  ilow 
metrics  (Henrv  &  Kafura)  useful’ 


/ 
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Goal  C2:  Select  a  set  of  dynamic  (test 
coverage  and  execution) 
metrics  for  the  APSE 

Goal  C2.1:  Develop  a  set  of  test  coverage 
metrics  for  the  APSE 

1)  Do  any  of  the  following  measures  of 
test  coverage  lead  to  a  useful  strategy 
for  testing:  number  of  statements 
executed?  number  of  decisions 
executed,  or  number  of  independent 
paths  executed? 

2)  Can  these  measures  be  extended  to 
provide  test  coverage  for  concurrent 
processing  or  will  new  measures  need 
to  be  developed?  Are  there  measures 
to  detect  starvation,  potential 
deadlocks,  etc.? 

3)  Are  there  other  features  of  .Ada  le.g.. 
exception  handling)  that  require  new 
measures  for  test  coverage?  What  are 
those  measures? 


Goal  C2.2:  Develop  a  set  of  execution 
metrics  for  the  APSE 

1)  What  are  useful  execution  metrics? 

2)  What  additional  information  do 
execution  statistics  provide  beyond 
what  can  be  gained  from  a  static 
view  of  the  system? 

3)  .Are  there  measures  of  execution 
complexity? 

4)  .Are  cenain  .Ada  features  or 
combinations  of  features  expanded 
into  verv  fast  or  very  slow  code? 

Goal  C3:  Develop  a  subjective  evaluation 
system  for  evaluating  some 
program  and  design  features 
that  are  not  easily  or 
practically  measured  in  other 
ways 

1)  Can  a  diverse  set  of  experts  i.Ada. 
applications,  and  methodologv 
experts)  accuratelv  evaluate  the 
subjective  aspects  of  the  project? 

2)  How  well  do  the  results  of  these 
evaluations  correlate  with  results 
from  objective  measures? 

3)  How  well  do  these  evaluations 
correlate  with  the  opinions  of  the 
development  team’ 

4 1  Can  we  conclude  anvthing  from  the 
subjective  results? 


The  categorization  of  the  metrics  used  on  this 
project  was  developed  by  Victor  Basil  1  and  other 
members  of  the  University  of  Maryland  team.  The 
software  development  methodology  employed 
benefited  from  their  direction.  The  authors  are 
grateful  to  Victor  Baslll,  John  Gannon, 
Elizabeth  Katz  and  Marvin  Zelkowitz  for  their 
help.  This  research  program  Is  monitored  by  the 
Office  of  Naval  Research  (ONR)  under  contract 
♦N00014-82-K-022S  to  the  University  of  Maryland 
with  funding  from  ONR  and  the  Ada  Joint  Program 
Office.  The  views  expressed  In  this  paper, 
however,  are  not  necessarily  those  of  the  Office 
of  Naval  Research,  the  Ada  Joint  Program  Office 
or  the  Department  of  Defense. 
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